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The  Honorable  John  McCain 
Chairman,  Committee  on  Commerce, 

Science,  and  Transportation 
United  States  Senate 

The  Honorable  Slade  Gorton 
Chairman,  Subcommittee  on  Aviation 
Committee  on  Commerce,  Science,  and 
Transportation 
United  States  Senate 

Previous  studies  of  aviation  safely  have  found  that  pilot  performance  is  a 
major  contributor  to  airhne  accidents  and  incidents  (events  that  affect  or 
could  affect  a  flight’s  safety).  Therefore,  training  to  improve  pilots’ 
performance  has  been  a  primary  effort  to  improve  airline  safety.  As  part  of 
this  effort,  some  airlines  have  provided  training  in  crew  resource 
management  (crm)  since  the  early  1980s,  and  the  Federal  Aviation 
Administration  (faa)  will  require  all  airlines  to  have  implemented  this 
training  for  pilots  by  March  1998.  crm  is  an  approach  to  improving  pilot 
performance  that  focuses  on  better  coordination — among  members  of  the 
cockpit  crew  as  well  as  among  the  cockpit  crew  and  flight  attendants, 
dispatchers,  and  air  traffic  controllers — ^to  handle  certain  routine  and 
emergency  situations. 

Airlines  can  meet  the  crm  training  requirement  in  one  of  two  ways:  (1)  by 
following  faa’s  traditional  requirements  for  training  pilots  and 
crew — ^specified  in  part  121  of  the  federal  aviation  regulations^ — or  (2)  by 
instituting  the  Advanced  Qualification  Program  (aqp),^  which  combines 
CRM  training  with  technical  training  for  pilots.  Part  121  training 
requirements  have  been  in  place  without  significant  modification  since  the 
1970s,  and  until  1990,  all  airlines  had  to  meet  these  requirements.  Since 
1990,  FAA  has  offered  airlines  aqp  training  as  an  alternative  to  traditional 
part  121  training,  and  eight  major  airlines  have  chosen  to  train  their  pilots 
imder  aqp  requirements. 

This  report  responds  to  your  request  that  we  examine  the  role  of  airUne 
pilots’  performance  in  accidents  and  faa’s  efforts  to  address  any 
inadequate  performance.  Specifically,  we  agreed  to  address  the  following: 


^14  C.F.R.  part  121,  subparts  N  and  O. 

^Special  Federal  Aviation  Regulation  No.  58 — Advanced  Qualification  Program. 
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(1)  What  are  the  types  and  frequency  of  accidents  in  which  an  airline 
pilot’s  performance  was  cited  as  a  contributing  factor,  including  those  in 
which  failme  to  use  crm  principles  was  identified,  and  (2)  how  adequate  is 
faa’s  gmdance  for  and  oversight  of  the  airlines’  implementation  of  pilots’ 
training  for  crm?  We  limited  our  review  to  the  accidents  and  incidents 
experienced  and  training  implemented  by  the  10  major  U.S. 
airlines — ^those  generating  $1  billion  or  more  in  revenues  annually.® 


Results  in  Brief 


Of  the  169  accidents  that  involved  the  major  airlines  and  that  were 
investigated  and  reported  on  in  detail  by  the  National  Transportation 
Safety  Board  from  1983  through  1995,  about  30  percent  were  caused  in 
part  by  the  pilots’  performance,  according  to  our  analysis.  In  at  least 
one-third  of  these  accidents  (about  16),  we  determined  that  the  pilots  did 
not  correctly  use  the  principles  of  crew  resource  management.  For 
example,  according  to  the  National  Transportation  Safety  Board,  just 
before  the  1994  crash  in  Charlotte,  North  Carolina,  which  killed  37  people, 
the  aircraft  had  encountered  a  sudden  change  in  wind  direction  and  the 
captain  gave  an  incorrect  order  to  the  first  officer,  who  did  not  question 
the  order,  as  crew  resomrce  management  principles  would  require. 
Furthermore,  during  the  same  period,  of  the  nearly  4,000  incidents,  we 
found  that  about  one-fifth  were  caused  in  part  by  the  pilots’  performance. 

faa’s  guidance  for  and  oversight  of  training  in  crew  resomrce  management 
does  not  ensure  the  adequacy  of  this  training  under  part  121,  while  they  do 
imder  the  new  Advanced  Qualification  Program,  faa’s  guidance  for  the 
implementation  of  the  Advanced  Qualification  Program  specifies  a  process 
for  curriculum  development  that  the  airlines  must  follow  in  order  to 
integrate  training  in  crew  resource  management  with  technical  flying 
skills.^  FAA  inspectors  overseeing  this  training  assess  the  curriculum  to  see 
if  faa’s  process  has  been  followed;  this  assessment  also  enables  them  to 
determine  whether  the  pilots’  training  under  this  curriculum  is  adequate. 

In  contrast,  although  faa  requires  airlines  to  teach  crew  resource 
management  in  their  traditional  part  121  training,  the  guidance  it  provides 
on  how  to  develop  the  curriculum  for  this  training  is  ambiguous  and  does 
not  provide  standards  that  inspectors  can  use  to  evaluate  airlines’  training 
in  crew  resource  management.  Because  the  Advanced  Qualification 
Program  training  generally  differs  from  traditional  part  121  training  in  how 


^hese  airlines  are  Alaska,  American,  America  West,  Continental,  Delta,  Northwest,  Southwest,  Trans 
World,  United,  and  US  Airways. 

^Initially,  crew  resource  management  was  known  as  “cockpit  resource  management”  and  referred  only 
to  individuals  on  the  flight  deck— that  is,  to  pilots  and  flight  engineers. 
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it  develops  a  curriculum  for  training  in  crew  resomce  management,  the 
guidance  for  this  training  in  the  Advanced  Qualification  Program  may  not 
be  applicable  to  training  for  crew  resource  management  under  part  121. 
Therefore,  faa  needs  to  develop  guidance  for  teaching  crew  resource 
management  imder  traditional  part  121  training.  Furthermore,  although  8 
of  the  10  mgjor  airlines  plan  to  train  aU  their  pilots  under  aqp,  the  need  for 
guidance  on  crew  resource  management  training  under  part  121 
remains — both  for  those  airlines  that  have  opted  not  to  enter  the 
Advanced  Qualification  Program  as  weU  as  for  those  that  participate  in  the 
program  but  wLQ  nonetheless  continue  to  have  some  of  their  pilots  trained 
imder  part  121  for  up  to  8  years  as  they  make  the  transition  to  the 
Advanced  Qualification  Program. 


Background 


Airline  travel  is  one  of  the  safest  modes  of  public  transportation  in  the 
United  States.  The  current  level  of  airline  safety  has  been  achieved,  in  part, 
because  the  airline  industry  and  government  regulatory  agencies  have 
implemented  rigorous  pilot  training  and  evaluation  programs.  The  mgjor 
airlines  have  training  programs  for  pilots  that  focus  on,  among  other 
things,  maintaining  flying  skills,  qualifying  to  fly  new  types  of  aircraft,  and 
acquiring  skUls  in  dealing  with  emergencies. 

faa’s  original  regulations  for  the  airlines’  general  training 
programs — ^referred  to  in  this  report  as  part  121 — ^spell  out  the  number  of 
hovus  of  training  required  in  particular  areas,  such  as  the  time  spent 
practicing  emergency  procedmes.  Effective  for  1996,  faa  instituted  a 
requirement  for  crm  training  under  part  121  that  states  the  following: 

“After  March  19, 1998,  no  certificate  holder  [airline]  may  use  a  person  as  a 
flight  crewmember,  and  after  March  19, 1999,  no  certificate  holder  may  use 
a  person  as  a  flight  attendant  or  aircraft  dispatcher  unless  that  person  has 
completed  approved  crew  resource  management  or  dispatcher  resource 
management  initial  training,  as  applicable,  with  that  certificate  holder  or 
with  another  certificate  holder.”®  faa  believes  that  this  training  should 
improve  flight  crews’  performance.® 


'14  C.F.R.,  section  121.404. 

®This  requirement  applies  to  all  airlines  operating  under  part  121  and  those  airlines  certified  under  part 
135  that  conduct  training  under  part  121.  Airlines  now  operating  under  part  121  use  aircraft  configured 
for  10  or  more  passengers.  New  rules  adopted  by  FAA  in  1995  require  certain  commuter  operators 
conducting  scheduled  operations  under  part  135  to  conduct  those  operations  under  part  121  beginning 
in  March  1997.  Included  were  those  airlines  conducting  scheduled  operations  carrying  passengers  with 
aircraft  configured  for  10  to  30  seats. 
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As  an  alternative  to  training  under  these  regulations,  airlines  may  apply  to 
participate  in  aqpJ  Eight  of  the  10  major  airlines  have  applied  to,  and  been 
approved  for  participation  in,  aqp.  Unlike  traditional  part  121  training,  aqp 
specifies  the  criteria  for  the  required  level  of  performance  in  certain  types 
of  maneuvers,  rather  than  hours  of  training,  and  it  integrates  crm  training 
with  technical  flying  skills.  The  airlines  are  expected  to  fully  implement 
AQP  over  a  period  of  time,  up  to  8  years.  Full  implementation  means  that 
the  airlines  have  trained  their  pilots  for  each  type  of  aircraft  they  fly. 
Training,  however,  occurs  only  after  the  airline  has  gone  through  three 
other  stages:  (1)  getting  approval  to  participate  in  the  program, 

(2)  developing  a  training  curriculum,  and  (3)  training  instructors. 
Continuing  crew  training,  the  last  stage,  is  to  occur  annually. 

Responsibility  for  aqp  and  traditional  part  121  training  rests  with  different 
FAA  branches.  The  aqp  Branch  within  the  Office  of  Flight  Standards 
Services  oversees  aqp,  and  the  Branch  e3q)ects  to  transfer  many  of  its 
oversight  responsibilities  to  inspectors  in  the  field  as  each  airline  fuUy 
implements  its  aqp.  The  administration  of  traditional  part  121  training  is 
divided  between  the  Air  Carrier  Training  Branch,  which  sets  training 
requirements,  and  the  flight  standards  inspectors  in  the  field,  who  are 
responsible  for  overseeing  the  training,  faa’s  inspectors  periodically 
review  and  approve  airlines’  curricula  and  training  materials  and  observe 
training. 

CRM  is  a  “human  factors”  approach  for  improving  aviation  safety  by 
preventing  or  managing  pilots’  errors.  Human  factors  refers  to  a 
multidisciplinary  effort  to  develop  information  about  human  capabilities 
and  limitations  and  to  apply  this  information  to  equipment,  systems, 
facilities,  procedures,  jobs,  environments,  training,  staffing,  and  personnel 
management  for  safe  and  effective  human  performance.  Under  this 
approach,  pilots  are  trained  to  recognize  potential  mistakes  in  judgment  or 
actions  and  to  compensate  for  them  to  prevent  accidents  and  incidents.^ 


"^FAA  issued  the  final  Special  Federal  Aviation  Regulation  (SFAR)  58  for  AQP  on  October  2, 1990  and 
the  termination  date  for  the  regulation  has  been  extended  to  October  2,  2000.  The  SFAR  58  is  found  in 
part  121.  The  AQP  advisory  circular  was  issued  on  August  9, 1991. 

®The  National  Transportation  Safety  Board  (NTSB),  the  official  source  of  information  on  airline 
accidents,  defines  accidents  as  occurrences  in  which  individuals  are  killed  or  suffer  serious  iiyuiy,  or 
the  aircraft  is  substantially  damaged.  By  NTSB's  definition,  accidents  can  range  from  fatal  crashes  in 
which  all  on  board  are  killed  to  events  in  which  only  one  person  suffers  a  broken  bone  and  the  aircraft 
is  not  damaged,  to  still  others  in  which  there  is  substantial  aircraft  damage  but  no  fatalities  or  serious 
iiyuries.  NTSB  generally  distinguishes  between  accidents  and  incidents.  NTSB  defines  incidents  as 
occurrences  other  than  accidents  associated  with  the  operation  of  an  aircraft  that  affect  or  could 
affect  the  safety  of  operations.  (49  C.F.R.  section  830.2) 
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For  example,  in  training  for  initial  departure,  crm  training  has  the  captain 
practice  briefing  the  crew  about  the  actions  to  be  taken  if  the  takeoff  must 
be  aborted  because  of  an  emergency,  crm  also  teaches  the  crew  to 
question  orders  when  they  believe  they  have  information  that  indicates 
these  orders  are  inappropriate.  Similarly,  crm  training  teaches  the  crew  to 
anticipate  problems  and  make  decisions  that  take  these  anticipated 
problems  into  account. 


Airline  Pilots’ 
Performance  Was  a 
Contributing  Factor  in 
Many  Accidents  and 
Incidents 


About  30  percent  of  the  169  accidents  and  18  percent  of  the  3,901  incidents 
that  occurred  from  1983  through  1995  were  caused  at  least  in  part  by 
pilots’  performance,  according  to  our  analysis  of  the  National 
Transportation  Safety  Board’s  (ntsb)  and  faa’s  data.  Furthermore,  the 
accident  data  indicate  that  nearly  one-third  of  the  accidents  occurred 
because  the  pilots  either  did  not  follow,  or  did  not  correctly  foUow,  crm 
principles.  The  most  frequently  occurring  accidents  and  incidents  included 
collisions  on  the  groimd  with  objects  and  other  airplanes,  flights  through 
turbulent  weather  that  resulted  in  iiyuries,  and  deviations  from  flight  paths 
that  had  the  potential  to  cause  an  in-flight  collision. 


Data  Show  Pilots’  On  the  ground,  pilot  performance  was  associated  most  frequently  with 

Performance  Contributed  airplanes  colhding  with  vehicles,  buildings,  other  equipment,  or  animals. 

to  Accidents  and  Incidents  ^  (32  percent!  and 

(34  percent).  Figure  1  shows  the  types  of  accidents  and  mcidents  on  the 

ground  reported  from  1983  through  1995. 
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Figure  1:  On-the-Ground  Accidents  and  Incidents  Associated  With  Pilots’  Performance 


Accidents^ 


Incidents’’ 


®NTSB  cited  62  events  associated  \A/ith  piiots'  performance  in  169  accidents. 

‘’FAA  cited  446  events  associated  w/ith  piiots'  performance  in  3,901  incident  reports. 


Sources:  FAA  and  NTSB. 

In  the  air,  pilot  performance  was  most  frequently  associated  with  injuries 
to  passengers  and  flight  attendants  during  turbulent  weather — 41  percent 
of  accidents  and  12  percent  of  incidents.  Figure  2  shows  the  types  of 
accidents  and  incidents  in  the  air  that  were  reported. 
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Figure  2;  In-the-Air  Accidents  and  incidents  Associated  With  Piiots’  Performance 


Accidents  ^  Incidents 


Note:  Amounts  may  not  add  to  100  percent  due  to  rounding. 


^NTSB  cited  46  events  associated  with  pilots'  performance  in  169  accidents. 
^FAA  cited  209  events  associated  with  pilots'  performance  in  3,901  incidents. 


Sources;  FAA  and  NTSB. 


In  addition  to  the  accidents  and  incidents  discussed  above,  faa  maintains 
data  separately  for  those  occasions  on  which  pilots  failed  to  comply  with 
the  air  traffic  controller’s  instructions — ^such  as  not  staying  on  the  directed 
flight  path  and/or  entering  a  runway  without  clearance.®  Of  the  1,471 
imauthorized  maneuvers  from  1987  through  1995,^®  80  percent  occurred  in 
the  air,  and  most  of  these  (73  percent)  occurred  when  pilots  did  not 
maintain  their  assigned  altitude  levels.  The  imauthorized  pilot  maneuvers 
on  the  ground  were  most  often  (69  percent)  associated  with  pilots’  moving 
airplanes  onto  runways  without  authorization  from  the  air  traffic  control 


^hese  data  are  found  in  FAA’s  pilot  deviation  database. 

^^ese  data  ’were  available  from  FAA’s  National  Aviation  Safety  Data  Analysis  Center  only  for  these 
years. 
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tower.  These  types  of  incidents  have  the  potential  to  cause  accidents.  For 
example,  the  December  1990  crash  at  the  Detroit  Metropolitan  Airport 
occurred  when  an  airplane  taxied  onto  a  runway  being  used  for  takeoff  by 
another  airplane  and  collided  with  that  airplane.  Twelve  people  died.  The 
first  plane  had  not  gotten  permission  from  the  control  tower  to  enter  this 
runway,  as  it  should  have.  Figure  3  shows  the  most  frequently  reported 
imauthorized  pilot  maneuvers  in  the  air  and  on  the  ground. 


Deficiencies  in  CRM  hi  our  analysis  of  accidents,  we  found  deficiencies  in  the  airline  pilots’  use 

rnntrihntpH  to  ArriHpnts  of  CRM  in  nearly  one-third  of  all  accidents  involving  pilots’  performance. 

Moreover,  we  found  crm  deficiencies  in  half  of  the  serious  accidents  in 
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which  there  was  at  least  one  fatality.  About  46  percent  of  these  crm 
deficiencies  involved  a  lack  of  coordination  among  members  of  the 
cockpit  crew,  as  well  as  the  captain’s  failure  to  assign  tasks  to  other  crew 
members  and  to  effectively  supervise  the  crew.  Generally,  these  crm 
deficiencies  illustrated  the  importance  of  effective  communication. 

For  example,  in  the  Charlotte,  North  Carolina,  crash  in  July  1994, 
communication  among  crew  members  did  not  occur,  according  to  ntsb’s 
accident  investigation  report,  ntsb  believes  that  the  captain,  who  was  not 
flying  the  aircraft  at  the  time  and  could  not  see  the  groimd  because  of  poor 
visibility,  became  disoriented  and  commanded  the  first  officer,  “down, 
push  it  down,”  even  though  they  were  encoimtering  windshear,  which  is  a 
sudden  change  in  wind  direction.  The  first  officer  did  not  question  the 
order,  as  he  should  have,  according  to  ntsb,  because  the  windshear  was 
creating  an  unstable  situation;  the  plane  could  not  recover  from  the 
sudden  downward  shift  in  direction  caused  by  following  the  captain’s 
order.  The  plane  crashed  nose  down  into  the  ground,  and  37  people  died. 

Similarly,  in  a  June  1984  accident  in  Detroit,  Michigan,  a  lack  of 
commurucation  between  the  crew  and  air  traffic  controllers  during  a 
landing  in  a  severe  thunderstorm  contributed  to  the  accident,  according  to 
the  NTSB  report.  The  crew  did  not  request  clarification  about  the  weather 
conditions  or  change  its  course  of  action  to  take  these  conditions  into 
account.  The  winds  associated  with  the  storm  forced  the  plane  down 
precipitously,  causing  an  emergency  landing  without  the  landing  gear’s 
being  fully  extended.  The  plane  skidded  off  the  runway,  causing  serious 
damage  to  the  aircraft  and  an  emergency  evacuation  of  the  passengers. 
NTSB  reported  that  the  lack  of  crm  practices  was  a  probable  cause  of  the 
accident. 

The  National  Aeronautics  and  Space  Administration  reported  similar 
results  in  its  analysis  of  pilot  reports  submitted  to  its  voluntary  reporting 
system.^  Nearly  half  of  the  reports  cited  deficiencies  in  the  pilots’  use  of 
CRM  principles;  about  53  percent  of  the  crm  deficiencies  concerned 
coordination  among  members,  assignment  of  tasks,  and  crew  supervision. 


Analysis  of  part  121  Crew  Resource  Management  Incidents,  National  Aeronautics  and  Space 
Administration,  Quick  Response  No.  296,  Aviation  Safety  Reporting  System  (Feb.  6, 1997). 
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FAA’s  Guidance  and 
Oversight  Do  Not 
Ensure  Effectiveness 
of  CRM  Training 


For  AQP  training,  faa  has  specified  the  process  airlines  need  to  follow  to 
develop  and  implement  a  curriculum  that  integrates  crm  concepts  with 
technical  flying  skUls,  but  faa’s  guidance  for  crm  training  under  part  121 
does  not  have  the  same  degree  of  specificity.  As  a  result,  inspectors 
overseeing  training  under  part  121  do  not  have  standards  they  can  use  to 
evaluate  airlines’  crm  training  curriculum  and  the  delivery  of  that  training. 
Generally,  inspectors  could  not  use  the  guidance  provided  under  aqp  to 
evaluate  part  121  training  for  the  crm  curriculum  because  the  cmricula 
developed  imder  the  two  programs  differ  significantly.  As  a  result,  airlines 
continue  to  need  specific  guidance  for  crm  xmder  part  121 — ^both  those 
airlines  that  have  opted  not  to  enter  aqp  as  well  those  that  will  continue  to 
train  at  least  some  of  their  crews  imder  part  121  until  they  have  fully 
implemented  aqp,  which  could  take  up  to  8  years. 


faa’s  Guidance  for  CRM 
TVaining  Is  Detailed  Under 
AQP  but  Not  Under  Part 
121 


Once  an  airline  elects  to  participate  in  aqp,  it  must  follow  SFAR  68  (the  aqp 
regulation)  for  developing  a  formal  curriculum — ^including  assessing  the 
sMUs  pilots  need  to  safely  operate  the  aircraft  they  fly,  developing 
curriculum  objectives  for  teaching  those  sMUs,  having  measurable  criteria 
for  evaluating  whether  the  pilots  have  achieved  those  objectives,  and 
developing  materials  to  teach  those  objectives,  faa  must  approve  this 
curriculum.  Furthermore,  aqp  requires  aU  airlines  to  train  their  pilots  in 
simulators  so  that  they  gain  experience  with  a  number  of  emergency 
situations.  Finally,  airlines  must  submit  data  to  faa  demonstrating  that 
their  crews  have  mastered  the  skills  they  need  to  fly  for  those  airlines.*^ 


In  developing  its  aqp  curriculiun,  an  airline  is  required  to  integrate  crm 
training  into  every  aspect  of  its  crews’  training.  As  a  result,  the  pilots 
trained  under  aqp  are  assessed  on  crm  principles  as  well  as  on  technical 
flying  skills.  For  example,  when  a  pilot  changes  the  aircraft’s  altitude — a 
technical  flying  skill — crm  principles  dictate  that  this  pilot  inform  the 
other  pilot  by  verbally  annoimcing  the  new  altitude  while  continually 
pointing  to  the  altitude  indicator  until  the  other  pilot  also  points  to  the 
altitude  indicator  and  repeats  the  new  altitude.  This  procedure  is  used  to 
ensme  that  neither  pilot  will  fail  to  maintain  the  appropriate  altitude. 

In  contrast,  faa’s  requirements  for  crm  training  imder  part  121  do  not 
require  airlines  to  develop  a  curriculum  for  crm  training  with  measurable 
criteria  or  to  integrate  that  curriculum  with  other  aspects  of  part  121 
training.  For  the  crm  curriculum  under  part  121,  faa  provides  suggested 


^^aining  records  for  pilots  are  maintained  under  part  121.  Performance  data  for  crews  submitted  to 
FAA  under  AQP  permit  the  agency  to  conduct  its  own  analyses  of  crews’  mastery. 
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training  topics  but  does  not  clearly  lay  out  how  the  airlines  are  to 
introduce  these  topics  into  their  training  programs,  according  to  airline 
officials  and  faa  inspectors.  For  example,  faa  recommends  that  airlines 
train  crews  in  “workload  management  and  situational  awareness.”  For  this 
training,  faa  suggests  such  topics  as  “preparation/planning/vigilance”  and 
“workload  distribution/distraction  avoidance.”  However,  for  those  airlines 
that  choose  to  integrate  these  topics  with  technical  flying  skills,  faa  does 
not  explain  how  the  airlines  are  to  do  so.^^ 

The  lack  of  specificity  in  faa’s  guidance  for  the  development  of  a  crm 
ciuriculum  under  part  121  contrasts  with  the  detailed  guidance  faa 
provides  for  the  development  of  a  curricidvun  on  technical  flying  skills. 

For  example,  faa’s  guidance  on  how  pilots  are  to  respond  to  windshear 
under  part  121  directs  them  in  a  number  of  technical  flying  sldlls,  such  as 
how  to  handle  the  rudder,  but  it  is  silent  on  how  to  employ  crm  principles 
in  this  situation.  In  contrast,  imder  aqp,  faa’s  guidance  instructs  the 
airlines  to  specify  not  only  the  technical  skills  but  also  the  crm  principles 
that  must  be  applied  in  a  windshear  situation. 

Because  faa’s  guidance  on  crm  training  imder  part  121  is  less  specific, 
airlines  vary  in  how  they  deliver  their  crm  training.  While  all  the  airlines 
provide  classroom  training  in  crm  principles  under  part  121  training,  they 
may  not  integrate  this  training  with  technical  flying  skills.  For  example, 
airlines  may  (1)  train  pilots  in  technical  flying  skills  in  flight  simulators 
without  integrating  crm  principles  or  (2)  integrate  crm  principles  with 
technical  flying  skills  in  flight  simulators.  Generally,  we  found  that  crm 
training  had  been  integrated  with  technical  flight  training  to  a  higher 
degree  at  those  airlines  that  were  in  later  phases  of  aqp  implementation. 


FAA’s  Oversight  of  CRM 
Training  Is  Adequate  for 
AQP  but  Not  for  Part  121 


In  developing  aqp,  faa  incoiporated  procedures  for  evaluating  crm  training 
and  developed  a  process  for  ensuring  that  faa  inspectors  would  have  the 
criteria  they  need  to  conduct  the  evaluations  for  pilots’  training  on 
different  types  of  aircraft.  Specifically,  aqp  provides  a  systematic  way  of 
identifying  the  tasks  and  subtasks  involved  in  a  particular  phase  of  flight. 
Therefore,  an  inspector  observing  the  training  program  can  determine 
whether  crm  principles  are  being  invoked  in  a  given  flight  situation.  For 
example,  when  a  crew  is  preparing  for  landing,  aqp  specifies  that  the  first 
officer,  if  unsure  of  the  planned  course  of  action  in  the  event  of  a  missed 
approach,  is  to  ask  the  captain  to  clarify  the  plan  so  that  both  have  a  full 
understanding  of  the  actions  they  will  take.  Similarly,  if  a  flight  has  to  be 


^^See  faa’s  Advisory  Circular  120-5 IB,  Crew  Resource  Management  Training. 
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diverted  from  one  airport  to  another,  the  captain  is  to  direct  the  first 
officer  to  (1)  get  out  the  maps  for  the  alternate  airport,  (2)  notify  the  flight 
attendants,  and  (3)  make  the  armoimcement  to  the  passengers.  This 
delegation  of  tasks  allows  the  captain  time  to  handle  radio  contact  with 
the  airline’s  dispatchers  and  air  traffic  controllers,  obtain  weather  updates 
at  the  alternate  airport,  and  fly  the  plane. 

In  the  early  stages  of  aqp  implementation,  the  aqp  Branch  is  evaluating 
airlines’  training,  faa  will  transfer  this  responsibility  to  inspectors  in  the 
field  as  airlines  fully  implement  aqp.  Field  inspectors  will  be  trained  in 
evaluating  the  crm  training  as  an  integral  part  of  their  evaluation  of  aqp 
training.  The  inspectors  at  those  airlines  that  had  progressed  beyond  the 
initial  phases  of  aqp  noted  that  they  had  received  aqp  training  at  the 
airlines  for  which  they  were  responsible.  Moreover,  all  of  the  inspectors 
we  spoke  with  maintained  that  while  certain  facets  of  aqp  were  fixed, 
some  parts  were  still  evolving.  As  a  result  of  the  program’s  flexibiUly  and 
evolution,  the  inspectors  pointed  out  that  it  was  not  possible  to  structure  a 
training  program  for  them  that  could  cover  every  aspect  of  aqp  at  every 
airline.  Despite  this  fluidity,  these  inspectors  said  that  the  aqp  Branch 
Office  made  sure  that  the  program’s  standards  were  maintained  across 
airlines. 

While  the  evaluation  of  the  delivery  of  crm  training  is  incorporated  into  the 
oversight  process  for  aqp  training,  it  is  not  under  traditional  part  121 
training.  Moreover,  faa  has  not  provided  its  inspectors  with  any  specific 
guidance  or  training  for  evaluating  airlines’  crm  training  imder  part  121. 
Although  FAA  inspectors  may  obtain  some  crm  training  from  a  3-hour 
computerized  interactive  course,  this  lack  of  guidance  for  evaluating  crm 
training  imder  part  121  is  troublesome  to  the  inspectors  we  spoke  with 
because  of  what  they  view  as  an  inherent  conflict  between  performance 
expectations  for  individuals  under  part  121  and  crew  performance 
expectations  articulated  in  crm  principles.*'*  Under  part  121,  phots  are  to 
master  technical  flying  skills  and  perform  these  skiUs  without  reliance  on 
any  other  crew  member,  ha  contrast,  crm  principles  and  training  teach 
pilots  how  to  use  to  maximum  effect  the  abilities  and  ejqperience  of  other 
crew  members,  as  well  as  their  own  technical  flying  skills. 

Without  formal  faa  instructions,  inspectors  have  developed  their  own 
approaches  to  this  evaluation.  For  example,  one  inspector  said  that  he 
based  his  approval  on  his  belief  that  the  airline  for  which  he  was 


'■•in  a  previous  review  of  FAA’s  training  for  its  inspector  workforce — Aviation  Safety:  Targeting  and 
Training  of  FAA’s  Safety  Inspector  Workforce  (GAO/r-RCED-96-26,  Apr.  30, 1996) — ^we  found  that 
some  inspectors  were  unaware  that  needed  training  was  available  through  computer-based  courses. 
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responsible  “had  a  good  safety  record”  and  “would  probably  establish  a 
good  program.”  Another  inspector  said  that  in  approving  any  training 
program,  he  sought  guidance  first  from  any  applicable  federal  aviation 
regulation;  the  Inspector’s  Handbook,  applicable  advisory  circulars;  and, 
finally,  any  other  faa  publication,  such  as  the  Introduction  to  crm  Training. 
However,  this  inspector  added  that  these  sources  did  not  provide  the 
criteria  he  needed  to  evaluate  crm  training.  As  a  result,  he  looked  for 
behaviors  such  as  crew  members’  “working  together”  to  resolve  problems, 
“catching  errors,”  or  “dealing  vdth  the  consequences  resulting  fi-om 
uncaught  errors.” 

These  ad  hoc  approaches  to  evaluating  the  deUvery  of  crm  training  are  not 
satisfactory  to  faa  officials  at  headquarters  or  to  officials  for  at  least  one 
airline,  faa  officials  told  us  that  the  agency  needed  additional  crm  training 
for  its  inspectors  conducting  reviews  under  part  121.  In  addition,  officials 
at  one  airline  told  us  that  the  lack  of  specific  guidance  and  training  for  faa 
inspectors  responsible  for  evaluating  crm  training  imder  part  121  has 
hampered  faa’s  ability  to  review  crm  programs.  Furthermore,  the  problems 
FAA  inspectors  face  in  evaluating  crm  training  under  part  121  will  continue 
indefinitely  in  the  absence  of  clearer  guidance  from  faa  for  those  airlines 
that  have  decided  not  to  enter  aqp  and  for  those  airhnes  in  the  program 
that  have  not  fuUy  implemented  it. 

Because  aqp  is  implemented  by  the  type  of  aircraft  the  crew  flies,  even  the 
airlines  that  have  been  accepted  for  aqp  will  continue  to  provide  some  crm 
training  imder  part  121.  For  the  eight  airlines  implementing  aqp,  we 
estimate  that  only  about  one-third  of  their  crews  have  begun  to  receive  aqp 
training.  Therefore,  most  crews  are  still  receiving  traditional  training 
under  part  121,  and  some  will  continue  do  so  for  up  to  8  years.  As  of 
September  1997,  the  airlines’  estimated  dates  for  completing  the  transition 
to  aqp  training  ranged  between  2000  and  2005.  (See  table  1.) 
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Table  1 :  Estimate  of  Completed  AQP 
Implementation  by  Airline 

Airline 

Year  begun 

Pilots  being 
trained  under 
AQP  as  of 
September  1997 

Estimated  year  of 
completion 

Alaska 

1994 

0 

2002 

American 

1994 

0 

2005 

Continental 

1997 

0 

2002 

Delta 

1992 

60% 

2001 

Northwest 

1994 

23% 

2001 

Trans  World 

1995 

0 

2000 

United 

1991 

50% 

2001 

US  Airways^ 

1994 

0 

2000 

^Estimate  is  for  one  aircraft  type.  US  Airways'  aircraft  fleet  composition  is  under  review,  and  no 
determination  has  been  made  to  place  any  other  aircraft  types  into  AQP  until  the  aircraft  fleet  plan 
is  resolved. 


Conclusions  pubUc,  safety  is  the  paramount  issue,  and  faa  and  the 

airlines  have  worked  to  provide  rigorous  training  programs  for  pilots. 
Crew  resource  management,  which  focuses  on  making  the  best  use  of  all 
available  experience  and  skills  in  the  cockpit,  is  increasingly  seen  as  an 
important  component  of  safe  flights,  faa  recognized  the  importance  of 
crew  resource  management  by  requiring  all  airlines  to  include  training  in 
these  principles  and  by  incorporating  crew  resource  management  into  its 
Advanced  Qualification  Program. 

Pilots’  performance  is  not  the  only  factor  in  airline  accidents,  but  it  is  an 
important  one.  We  identified  pilots’  performance  as  the  cause  of  about 
one-third  of  all  the  accidents  and  nearly  one-fifth  of  the  incidents  for  the 
10  mjgor  airlines  from  1983  through  1995. 

Training  for  safer  performance  by  pilots  that  teaches  crew  resource 
management  can  occur  irnder  either  the  Advanced  Qualification  Program 
or  part  121.  However,  while  faa’s  guidance  for  the  implementation  of  the 
Advanced  Qualification  Program  specifies  a  process  for  cmriculum 
development  that  integrates  this  training  with  training  in  technical  flying 
skills,  faa’s  guidance  for  curriculum  development  under  part  121  is 
ambiguous  and  does  not  provide  standards  that  inspectors  can  use  to 
evaluate  and  approve  airlines’  training  in  crew  resource  management.  As  a 
result,  FAA  cannot  be  assured  that  airlines  are  developing  a  curriculum  for 
teaching  crew  resource  management  that  wiU  effectively  teach  pilots  how 


Page  14 


GAO/RCED-98-7  Human  Factors 


to  best  use  all  the  skUls  and  experience  available  to  them  in  the  cockpit. 
Furthermore,  without  specificity  in  the  development  of  training  for  crew 
resource  management  under  part  121  and  without  any  guidance  on  how  to 
evaluate  this  training  imder  part  121,  faa  inspectors  are  relying  on  their 
own  experience  in  observing  pilots  or  even  on  the  belief  that  the  airline 
“would  probably  establish  a  good  program.”  These  problems  are  especially 
troublesome  because  pilots  who  have  not  completed  FAA-approved  crew 
resource  management  training  by  March  1998  may  not  fly  for  airlines. 

Recommendations 

To  help  ensure  that  airlines  appropriately  train  pilots  in  crm  principles 
under  part  121  and  that  faa  inspectors  are  able  to  imiformly  evaluate  this 
CRM  training,  we  recommend  that  the  Secretary  of  Transportation  direct 
the  Administrator  of  faa  to  develop  a  process  that  airlines  must  follow  for 
creating  a  crm  curriculum,  with  measurable  criteria,  imder  part  121  as  it 
has  for  the  Advanced  Qualification  Program. 

Agency  Comments 
and  Our  Evaluation 


with  the  Deputy  Associate  Administrator  for  Regulation  and  Certification, 
the  Deputy  Director  of  Flight  Standards  Services,  the  Managers  for  the  Air 
Carrier  Training  Branch  and  the  Advanced  Qualification  Program,  and 
other  officials,  faa  commended  our  review  of  crm  training  at  the  nation’s 
airlines,  faa  accepted  the  report’s  recommendation  in  part,  faa  agreed  that 
it  should  ensiue  that  pilots  are  appropriately  trained  and  noted  that  crm 
training  can  provide  desirable  consequences  in  aviation  safety.  It  further 
agreed  that  uniform  evaluation  of  crm  training  using  measurable  criteria  is 
a  commendable  objective.  However,  faa  stated  that  science  has  not  yet 
developed  valid,  reliable  criteria  for  measuring  crm  performance,  faa  also 
agreed  that  more  can  be  done  to  develop  a  process  that  airlines  and 
inspectors  can  follow  to  create  a  crm  curriculum,  faa  indicated  that  better 
guidance  would  be  provided  in  a  number  of  ways,  such  as  updating 
Advisory  Circular  120-51,  Crew  Resoiuce  Management  Training,  and 
supplemental  guidance  for  inspectors  included  in  the  inspectors’ 
handbook  and  holding  regional  meetings  with  crm  specialists  from  Flight 
Standards  Services  and  other  organizations. 


We  concur  with  faa  that  crm  training  for  pilots  could  improve  aviation 
safety.  However,  we  believe  that  before  the  contribution  of  crm  training  to 
aviation  safety  can  be  measured,  it  is  necessary  to  determine  the  extent  to 
which  the  delivery  of  crm  training  for  pilots  has  occurred.  We  further 
concur  with  faa  that  more  should  be  done  to  develop  processes  for 
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airlines  and  inspectors  to  follow  in  creating  a  crm  curriculum.  We  believe 
that  imtil  faa  establishes  a  process  for  crm  curriculxun  development  that 
includes  an  assessment  of  the  extent  to  which  pilots  have  mastered  that 
curriculum,  it  will  not  be  possible  to  measiue  crm’s  performance  in 
contributing  to  aviation  safety. 


Scope  and 
Methodology 


To  determine  the  extent  to  which  inadequate  performance  by  pilots  was  a 
problem  for  the  10  major  U.S.  airlines,  we  examined  the  types  and 
frequency  of  safety-threatening  events — ^incidents  and  accidents — ^from 
1983  through  1995. 

To  determine  the  adequacy  of  faa’ s  guidance  for  and  oversight  of  pilots’ 
training,  we  reviewed  faa’s  role  in  the  airlines’  implementation  of  crm.  We 
focused  primarily  on  crm  training  because  faa  has  described  the  failure  to 
apply  CRM  principles  as  a  more  important  contributing  factor  in  accidents 
than  technical  flying  skills.  We  also  compared  faa’s  rules  and  regulations 
and  other  guidance  for  crm  training  with  that  provided  for  other  training 
programs,  as  well  as  interviewed  faa  and  airline  officials.  A  detailed 
discussion  of  our  methodology  is  presented  in  appendix  I.  Related  gao 
products  are  listed  at  the  end  of  this  report. 

Our  work  was  performed  from  October  1996  through  October  1997  in 
accordance  with  generally  accepted  government  auditing  standards. 


As  arranged  with  your  offices,  unless  you  publicly  annoxmce  its  contents 
earlier,  we  plan  no  further  distribution  of  this  report  imtil  30  days  after  the 
date  of  this  letter.  At  that  time,  we  will  provide  copies  of  the  report  to  the 
Secretary  of  Transportation,  appropriate  congressional  conunittees,  and 
other  interested  parties.  We  wiU  also  make  copies  available  to  others  upon 
request. 
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Please  contact  me  at  (202)  512-3650  if  you  or  your  staff  have  any  questions 
about  this  report.  Major  contributors  to  this  report  are  listed  in  appendix 

n. 


Gerald  L.  Dillingham 
Associate  Director,  Transportation 
Issues 
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To  identify  the  types  and  frequencies  of  accidents  and 
incidents — safety-threatening  events — related  to  pilot  performance,  we 
reviewed  accident  and  incident  data,  including  pilot  deviations,  contained 
in  the  National  Transportation  Board’s  (ntsb)  and  the  Federal  Aviation 
Administration’s  (faa)  electronic  databases.  We  obtained  these  data  from 
faa’s  National  Aviation  Safety  Data  Analysis  Center.  We  limited  our  review 
to  the  reported  events  in  accident  and  incident  data  sources  involving  the 
10  major  U.S.  passenger  airlines  from  1983  through  1995.  We  did  not 
independently  verify  these  data. 

To  facilitate  the  comparison  of  accidents  with  incidents  in  our  analysis  of 
the  types  and  frequencies  of  safety-threatening  events,  we  made  two 
ac^ustments  to  the  data.  First,  because  of  differences  in  the  way 
information  is  recorded  in  these  databases,  we  matched  the  similar 
categories  contained  in  both  databases  and  used  these  categories  in  our 
analysis.  For  example,  both  ntsb’s  and  faa’s  databases  contain  the 
category  “on  ground  collision  with  object,”  which  means  an  airplane 
struck  an  object,  such  as  a  vehicle  or  structure,  while  moving  on  the 
grotmd.  Second,  because  the  occurrences  of  events  in  accidents  closely 
conform  to  those  in  incidents,  we  used  the  events  that  occmred  in  each  of 
the  169  accidents  as  our  unit  of  analysis.  In  our  analysis  of  crew  resource 
management  (crm)  deficiencies,  we  used  the  accident  as  the  unit  of 
analysis  because  ntsb’s  findings  of  crm  deficiencies  were  by  accident  and 
not  by  the  individual  events  that  occurred  within  accidents. 

To  characterize  the  prevalence  of  pilot  performance  as  a  factor  in 
safety-threatening  events  over  time  and  between  airlines,  we  examined 
faa’s  incident  and  pilot  deviation  databases.  We  used  these  two  databases 
because  they  are  the  only  such  sources  with  adequate  nmnbers  of 
observations  to  make  such  comparisons. 

To  determine  the  extent  to  which  the  inadequate  use  of  crm  by  pilots 
contributed  to  accidents  and  incidents,  we  performed  a  content  analysis  of 
the  textual  information  foimd  in  the  factual  reports,  briefs,  and  final 
reports  of  the  169  accidents  investigated  by  ntsb  from  1983  through  1995. 
We  then  classified  crm  deficiencies  according  to  the  classification 
framework  presented  at  a  National  Aeronautics  and  Space  Administration 
(NASA)/Ames  workshop  in  1980.^®  This  framework  groups  crm  issues  into 
five  broad  clusters: 


^^Muiphy,  M.R.  (1980).  “Analysis  of  Eighty-four  Commercial  Aviation  Incidents:  Implications  for  a 
Resource  Management  Approach  to  Crew  Training.”  1980  Proceedings  Annual  Reliability  and 
Maintainability  Symposium.  Ames  Research  Center,  NASA,  Moffet  Field,  California. 
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•  (1)  Resource  management — ^the  application  of  specialized  cognitive  skills 
to  effectively  and  efficiently  utilize  available  resomces,  such  as  the  ability 
to  plan,  organize,  and  communicate. 

•  (2)  Organization  processes — crew  members’  actions  and  behaviors  in  the 
context  of  their  assigned  duties  and  expected  responsibilities. 

•  (3)  Personal  factors — ^the  knowledge,  skills,  abilities,  and  limitations  that 
individual  crew  members  bring  with  them  to  the  cockpit. 

•  (4)  Material  resources  internal  to  the  aircraft — ^the  cockpit  crew’s 
appropriate,  effective,  and  efficient  use  of  instructional  items,  such  as 
checklists,  and  navigational  charts  and  equipment,  such  as  on-board 
weather  radar,  navigational  controls,  and  engine  fire  extinguisher. 

•  (5)  Resources  external  to  the  aircraft — ^those  people  (air  traffic 
controllers),  entities  (airports),  and  circumstances  (emerging  poor 
weather)  that  may  affect  pilots’  plans,  decisions,  and  actions. 

Table  I.l  shows  the  classification  firamework  used  to  categorize  crm 
issues. 


Table  1.1 :  Classification  Framework  for  Categorizing  CRM  issues 

Clusters 

Material 

Resource 

management 

Organization 

processes 

Personal  factors 

resources 
internal  to  the 
aircraft 

Resources  external 
to  the  aircraft 

CRM  issues  -Social 

•Communication 

•Leadership 

•Management 

•  Planning 

•Problem-solving 

•Decision-making 

•Role 

•Monitoring 

•Workload 

•Knowledge 

•Proficiency 

•Experience 

•Motivation 

•Stress 

reaction 

•Fatigue 

•Textual 

•Equipment 

•Human 

•Facility 

•Environment 

To  verify  the  results  of  our  content  analysis,  we  requested  a  similar 
analysis  by  NASA’s  Aviation  Safety  Reporting  System  (asrs)  staff  of 
voluntarily  submitted  pilot  reports  contained  in  the  asrs  database. 
According  to  the  aviation  experts  we  consulted,  asrs  incident  reports 
provide  the  best  somrce  of  information  on  deficiencies  in  crm. 
Furthermore,  because  asrs  staff  are  most  familiar  with  the  data  and  have 
expertise  in  analyzing  this  free-form  data,  we  concluded  that  it  was  more 
appropriate  for  them  to  perform  this  analysis. 

To  evaluate  the  adequacy  of  faa’s  oversight  of  airline  pilot  training,  we 
obtained  faa’s  training  policies,  requirements,  guidance,  and  handbooks 
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relevant  to  crm  training.  We  discussed  training  programs,  including  CRM, 
and  training  procedures  with  appropriate  faa  officials,  including  officials 
in  the  Office  of  System  Safety,  the  Office  of  Regulation  and  Certification’s 
Flight  Standards  Services,  the  Advanced  Qualification  Program  Branch, 
the  Office  of  Accident  Investigation,  and  the  Human  Factors  Division.  In 
addition,  we  discussed  airline  training  evaluation  and  approval  processes 
and  obtained  training  documents  firom  faa  inspectors  responsible  for 
monitoring  airline  training.  Finally,  we  contacted  safety  directors  and 
trainers  at  the  major  airlines  and  obtained  documents  on  their  policies, 
procedures,  research,  and  training  curricula. 

We  requested  comments  fi'om  recognized  experts  in  the  field  of  human 
factors  in  academia  and  the  aviation  industry,  pilots,  and  government 
officials  from  faa,  ntsb,  and  nasa.  We  incorporated  their  comments  where 
appropriate  and  made  ac^ustments  to  our  methodology  as  warranted. 
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